Accelerating Iterations Involving Eigenvalue or Singular Value Decomposition by Block Lanczos with Warm Start

Many machine learning problems are solved by algorithms that involve eigenvalue decomposition (EVD) or singular value decomposition (SVD) in each iteration. Therefore, these algorithms suffer from the high computation cost of multiple EVD/SVDs. To relieve this issue, we introduce the block Lanczos method to replace the original exact EVD/SVD in each iteration by solving it approximately, yet still at a high precision. We also propose to utilize the subspace obtained in the previous iteration to start the block Lanczos procedure. Examples of three popular problems are presented to show how our block Lanczos with warm start (BLWS) technique can be adopted to accelerate its host algorithms. Experimental results show that our BLWS technique usually accelerates its host algorithms by at least two times.